Creating a custom three-dimensional body shape model

ABSTRACT

A series of captured images of a user is received. Using a processor, the images are processed to identify a portion of each of the images corresponding to the user. Parameters of a predetermined three-dimensional human model are modified to fit a modified version of the predetermined three-dimensional human model across the identified portions of the images to determine a set of specific parameters representing a body profile of the user.

BACKGROUND OF THE INVENTION

Achieving proper clothing fit can be difficult. Sizing measurements forclothing and body sizes are typically limited to only a few parameters.For example, conventional shirt and torso sizing may span a limitednumber of discrete sizes ranging from extra-small to extra-large. Theselimited sizes lack precision. Two shirts (or bodies) with a medium sizecan be drastically different. More detailed measurements are possibleand typically require a tailor to extensively measure a customer'sanatomy. This is both time-consuming and tedious. And although atailor's measurements can be accurate, clothing options are limitedsince it is difficult to identify readily available clothing that fitsto the tailor's measurements. Instead, clothing must typically be custommade or individually altered to match these measurements. Therefore,there exists a need for a more accurate method for measuring clothingand body sizing that is applicable across a large variety of clothingoptions and body shapes.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are disclosed in the followingdetailed description and the accompanying drawings.

FIG. 1 is a flow chart illustrating an embodiment of a process forcreating a custom three-dimensional model of a customer.

FIG. 2 is a flow chart illustrating an embodiment of a process forcreating a custom three-dimensional model of a customer.

FIG. 3 is a flow chart illustrating an embodiment of a process forcreating a custom three-dimensional model.

FIG. 4 is a flow chart illustrating an embodiment of a process forfitting a three-dimensional model to a body shape.

FIG. 5 is a diagram illustrating an example configuration for capturingimages of a customer's body shape.

FIG. 6 is a diagram illustrating an example image captured for creatinga custom three-dimensional model of a customer.

FIG. 7 is a diagram illustrating an example image annotated with jointdata for creating a custom three-dimensional model of a customer.

FIG. 8 is a diagram illustrating an example of a segmented image usedfor creating a custom three-dimensional model of a customer.

FIG. 9 is a diagram illustrating an example of a silhouette image usedfor creating a custom three-dimensional model of a customer.

FIG. 10 is a diagram illustrating an embodiment of a three-dimensionalhuman model fit to joint data.

FIG. 11A is a diagram illustrating an embodiment of a three-dimensionalhuman model compared to an image silhouette.

FIG. 11B is a diagram illustrating an embodiment of a process formodifying a three-dimensional human model to fit an image silhouette.

FIG. 12 is a diagram illustrating an embodiment of a process for fittinga three-dimensional human model to a customer body shape.

FIG. 13 is a diagram illustrating an embodiment of a graphical userinterface (GUI) for creating a three-dimensional human model of acustomer.

FIG. 14 is a diagram illustrating an embodiment of a graphical userinterface (GUI) for modifying body shape parameters of athree-dimensional human model.

FIG. 15 is a functional diagram illustrating a programmed computersystem for creating a custom three-dimensional model of a customer.

DETAILED DESCRIPTION

The invention can be implemented in numerous ways, including as aprocess; an apparatus; a system; a composition of matter; a computerprogram product embodied on a computer readable storage medium; and/or aprocessor, such as a processor configured to execute instructions storedon and/or provided by a memory coupled to the processor. In thisspecification, these implementations, or any other form that theinvention may take, may be referred to as techniques. In general, theorder of the steps of disclosed processes may be altered within thescope of the invention. Unless stated otherwise, a component such as aprocessor or a memory described as being configured to perform a taskmay be implemented as a general component that is temporarily configuredto perform the task at a given time or a specific component that ismanufactured to perform the task. As used herein, the term ‘processor’refers to one or more devices, circuits, and/or processing coresconfigured to process data, such as computer program instructions.

A detailed description of one or more embodiments of the invention isprovided below along with accompanying figures that illustrate theprinciples of the invention. The invention is described in connectionwith such embodiments, but the invention is not limited to anyembodiment. The scope of the invention is limited only by the claims andthe invention encompasses numerous alternatives, modifications andequivalents. Numerous specific details are set forth in the followingdescription in order to provide a thorough understanding of theinvention. These details are provided for the purpose of example and theinvention may be practiced according to the claims without some or allof these specific details. For the purpose of clarity, technicalmaterial that is known in the technical fields related to the inventionhas not been described in detail so that the invention is notunnecessarily obscured.

A technique to accurately and efficiently measure a body shape isdisclosed. Using images of a user captured from a device, such as asmartphone camera, a three-dimensional human model is created. In someembodiments, the three-dimensional human model functions as a virtualbody (or dress) form for rendering the fit of garments on a highlyaccurate model of the user. In some embodiments, the customthree-dimensional human model is generated by modifying parameters of apredetermined three-dimensional human model. The parameters are modifieduntil the custom three-dimensional human model accurately represents thebody profile of the user. For example, joints of the user are identifiedfrom images taken of the user and matched to joints in thethree-dimensional model. Similarly, silhouettes extracted from imagestaken of the user are matched to rendered silhouettes of the generatedthree-dimensional model. Matching both joints and silhouettes results isa custom three-dimensional model based on the user's body profile. Oncegenerated, a set of specific parameters that represent the body profileof the user can be extracted. This vector of parameters uniquely andefficiently defines the user's body profile and can be used for fitdecisions, style decisions, matching the user with stylists, and/orinventory allocation and purchasing decisions, among other clothing andinventory related decisions.

In some embodiments, a series of captured images of a user is received.For example, a user captures a series of images of the user from acamera such as the user's smartphone camera. As another example, aseries of images is extracted from video captured of the user. Theseries of images includes different poses of the user and captures theuser from different profiles. In some embodiments, the images areprocessed using a processor to identify a portion of each of the imagescorresponding to the user. For example, the images are segmented todifferentiate the foreground from the background, with the foregroundportions corresponding to the user. In some embodiments, thesegmentation process utilizes color information from the capturedimages. In some embodiments, a depth sensor is utilized and the capturedimages include corresponding depth information. The depth information isused to help identify the portion corresponding to the user. In someembodiments, parameters of a predetermined three-dimensional human modelare modified to create a modified three-dimensional human model thatfits the identified portions of the images corresponding to the user.For example, a predetermined three-dimensional human is used as atemplate to create a custom three-dimensional human model matching theuser's body shape as determined from the captured images. In someembodiments, the modified model is created by matching silhouettes ofthe modified model to silhouettes of the identified user portions of theimages. When the silhouettes match, the modified model accuratelyrepresents the user's body shape. In some embodiments, one step increating the matching modified model is matching joint locations of themodel to joint locations identified using the user portions of theimages. In various embodiments, the custom model is created by modifyingspecific parameters that are applied to the predeterminedthree-dimensional human model. These parameters modify the model'sheight, width, leg length, width at hips, etc. The selection of thesefeature parameters may be optimized to minimize overlap betweenparameters and to reduce the total number of parameters required todescribe a body shape. Once a modified model is created, a set ofspecific parameters representing a body profile of the user isdetermined. For example, the specific parameters used to modify themodel are also used to represent the body profile of the user. In someembodiments, the parameters are stored as a client vector representingthe user's body profile. The creation of a client vector has numerousapplications. The applications include but are not limited to matchinggarments to fit a client, visualizing garments on a virtualrepresentation of a client, matching stylists to clients with certainclient vector characteristics, informing inventory allocation andpurchases, and/or designing new garments to optimize the fit forpotential clients.

FIG. 1 is a flow chart illustrating an embodiment of a process forcreating a custom three-dimensional model of a customer. Using theprocess of FIG. 1, a custom three-dimensional model of a customer iscreated that accurately represents the body shape of the customer. Themodel is created in part from images captured of the customer. Forexample, customers can capture images or video of themselves using acamera such as a smartphone camera, tablet camera, or webcam. Theprocess of FIG. 1 may be performed in part by using a mobile deviceand/or remote processing server. In some embodiments, the process ofFIG. 1 is performed at least in part by using processor 1501 of computersystem 1500 of FIG. 15 and images captured via camera 1515 of computersystem 1500 of FIG. 15.

At 101, customer data is collected. For example, image data is capturedof the customer using a camera such as a smartphone camera. The imagedata may be processed by the smartphone and/or transmitted and collectedat a remote server for processing. In some embodiments, the image datais collected as video data. In various embodiments, different poses andbody profiles of the customer are collected. Along with image data, insome embodiments, additional customer data is collected such as sizingmeasurements. Sizing measurements may include measurements such as shoesize, shirt size, torso, waist, inseam, and/or bust, among others. Theadditional customer data may be optionally collected to augment thecollected image data. In various embodiments, customer information suchas name, age, gender, sex, style preferences, price preferences,delivery options, etc. may be collected as well.

At 103, a three-dimensional model of the customer is created. Using thecustomer data collected at 101, in particular the image data collected,a three-dimensional model of the customer is created. Thethree-dimensional model is a custom model that accurately represents thebody shape and profile of the customer. In some embodiments, the modelis created by starting with a predetermined three-dimensional humanmodel and modifying model parameters to fit the model to the customer'sbody shape. The modifications may be based on matching joint locationsand the customer's silhouette from the captured images. In someembodiments, the customer can additionally manipulate the model torefine the accuracy of the model. For example, the customer can adjustthe height, arm length, torso, etc. of the model to fit the customer'sbody profile, as appropriate. In some embodiments, the customer ispresented with the generated model, for example, overlaid on thecustomer's image. The customer can analyze and review the accuracy ofthe model and make/suggest changes. In some embodiments, the customercan submit additional images highlighting areas that need refinement.For example, in the event the feet are not accurately modeled, thecustomer can submit additional images that include focused images of thecustomer's feet that can be used to refine the accuracy of thecustomer's model.

At 105, the created three-dimensional model of the customer is appliedto one or more applications. In various embodiments, a set of specificparameters representing a body profile of the user is extracted from thecreated three-dimensional model of the customer. These parameters form acustomer or client vector describing the customer's body shape. Theclient vector can be used for a variety of applications includingidentifying and recommending for purchase potential garments that fitwell for the customer. For example, a value identifying a predicted sizefit can be determined for a clothing item based on the customer's clientvector. Using the client vector, the points of measure of the clothingitem can be compared to the customer's body shape. A higher value forthe predicted size fit indicates a higher likelihood the item fits thecustomer well and increases the probability the customer will purchasethe item. In some embodiments, items with high values for the predictedsize fit are provided to the customer to evaluate. For example, onlyitems with a value identifying the predicted size fit that exceeds aminimum threshold value are provided to the customer for evaluation. Thecustomer can rent and/or purchase the item after evaluating them. Theclient vector can also be used to identify and match particular styliststhat have a history of successful results with customers with similarclient vector profiles. For example, certain stylists are more aware ofand may have more experience with the fit challenges faced by clientswith particular client vector profiles. Certain stylists may also havebetter success metrics with customers with particular client vectors(e.g., finding well matched clothing). Additional applications includeusing client vectors for inventory purchasing decisions. For example,clothing items, lines, and/or brands that fit well based on clientvector matches are identified and targeted for purchasing decisionswhile those that do not fit well may be avoided. As another application,clothing can be manufactured based on client vectors. Clothing can bedesigned and made for clients with client vectors whose demands are notwell met by existing inventory options. In various embodiments, theapplications can be applied to individual client vectors and/or clustersof client vectors. For example, client vectors are clustered and itemsare designed and made based on the body shape of the cluster.

In some embodiments, the customer's three-dimensional model is used topredict the customer's traditional points of measurement. For example,measurements made using the model can predict the measurements that atailor would have taken by measuring the actual body of the customer.Instead, a virtual tape measure can be applied to the generated model topredict the same measurements.

In some embodiments, the customer's three-dimensional model is used asan avatar to visually demonstrate how clothing would appear on thecustomer. The accurately generated model allows the customer to see howdifferent garments would fit including how well different materials andstyles would drape on the customer's individual body shape. Athree-dimensional model also allows the user to view the garments fromall directions. The custom model also allows a customer to share arendered model with an outfit combination with others, for example, toget opinions from friends and family on purchasing decisions or to sharethe excitement of a recent purchase. The ease of viewing differentclothing items on the costumer's body greatly improves the customerpurchasing experience. Customers can virtually try on many moredifferent garments than would otherwise be possible.

FIG. 2 is a flow chart illustrating an embodiment of a process forcreating a custom three-dimensional model of a customer. Using theprocess of FIG. 2, customer data is collected for generating a customthree-dimensional model of a customer. In some embodiments, the data iscollected from a mobile device such as a smartphone device, a tablet, akiosk, a laptop, a smart television, and/or another similar device. Themodel may be created using the device for collecting the customer dataand/or a remote server. In some embodiments, the steps of 201, 203,and/or 205 are performed at 101 of FIG. 1, the steps of 207 and/or 209are performed at 103 of FIG. 1, and/or the step of 211 is performed at105 of FIG. 1.

At 201, customer measurements are received. For example, traditionalmeasurement sizes such as shirt size, shoe size, inseam, waist, dresssize, etc. may be received. Additional customer measurements may includemeasurements such as age, weight, fitness level, amount of muscle, etc.In various embodiments, the received customer measurements are used tohelp seed or augment the three-dimensional model of the customer. Insome embodiments, the measurements are received via a graphical userinterface (GUI) of a smartphone application. The measurements may alsobe received by analyzing past purchases made by the customer. Forexample, past garments are used to determine the customer's sizemeasurements. Both well fitting and poor fitting garments can be used todetermine and/or approximate the customer's measurements. A poor fittingpair of pants with customer feedback that they are too short can be usedto approximate leg length despite not fitting well.

At 203, customer images are captured. Using a camera, images of the userare captured. In some embodiments, a variety of poses are needed tocapture enough data of the customer. For example, a customer isinstructed to position her or himself in a predefined number ofdifferent poses. In some embodiments, the poses are selected to minimizeocclusions. For example, the arms are outstretched to prevent the armsfrom covering the torso. In some embodiments, the user is instructed tospin in a circle to capture different profiles of the user. The imagesmay be captured as individual images or captured as one or more videosequences of images. In various embodiments, the camera captures colorimages. In some embodiments, the camera is a three-dimensional cameraand captures depth information in addition to color information. In someembodiments, a separate depth sensor may be used to capture depth datathat is then merged with the camera image data.

In some embodiments, one or more predefined movements corresponding todifferent poses are captured. A predefined movement may include poses,one or more dance moves, a simulated runway walk, a curtsy, etc. Themovements may be selected from common movements but used to emphasizeseparation of the limbs from the body, the body from the background,and/or the identification of joints.

At 205, customer images are processed. In various embodiments, thecustomer images are processed to prepare the image data for creating athree-dimensional model. For example, individual images may be extractedfrom a video. In various embodiments, certain frames are cut anddiscarded, such as early frames and/or late frames in the video or imagesequence. For example, early and/or late frames may correspond to thecustomer preparing for a pose. In various embodiments, the imagescorresponding to required poses are determined and selected. The imagesmay be processed to remove noise, improve white balance, adjust forcolor contrast, etc. In some embodiments, multiple images are stitchedtogether. In some embodiments, the images are compressed. For example,in some embodiments, the customer images are transmitted to a remoteserver for processing and may be compressed prior to transmittal.

In some embodiments, the images are calibrated and/or annotated withmetadata. For example, the images can be calibrated using camera and/orsensor data such as lens data and gyroscope data. The zoom settingand/or orientation of the camera may be included with the image to helporient the image with respect to the customer and/or with respect toother images in the sequence. In some embodiments, a common origin, suchas a three-dimensional origin, is determined for each selected image andused to orient each image. In some scenarios, a close up image of aportion of the body may be difficult to place relative to other imageswithout origin data.

At 207, a three-dimensional human model of the customer is created.Using the images processed at 205, a three-dimensional human modelapproximating the customer's body shape is created. In some embodiments,the selected images may be segmented to identify the portion of the userfrom the image and a silhouette of the user portion is created. A modelof the user is created by comparing silhouettes of the model tosilhouettes extracted from the images. The model may be iterativelymodified until there is a match in silhouettes. In various embodiments,joint locations are identified from user portions of the images andmatched to joint locations in the model to improve the match.

In some embodiments, the three-dimensional human model is created usinga machine learning model. Using images of the user as input, a trainedmachine learning model is used to infer the model parameters needed tomodify a predetermined three-dimensional human model to fit thecustomer's body shape. The generated model can be compared for accuracyby comparing silhouettes of the model to silhouettes of the user portionof the image.

At 209, a determination is made whether additional data is needed. Inthe event no additional data is needed, processing continues to 211. Inthe event additional data is needed, processing continues back to 203.

In some embodiments, the accuracy of the generated model is determinedby comparing silhouettes of the generated model to silhouettes of theuser portion of the image. A determination is made whether the accuracyof the model is sufficient to accurately represent the customer's bodyshape. For example, an accuracy percentage is determined for the createdmodel. A determination is made whether the accuracy percentage meets athreshold value. In the event the accuracy percentage meets thethreshold value, no additional data is needed. In the event the accuracypercentage does not meet the threshold value, additional data is neededto refine the model. In some embodiments, an accuracy percentage isdetermined based on and associated with specific portions of the bodyshape. For example, a torso may be sufficiently accurate but the leftfoot may not be sufficiently accurate and may require additional data.As another example, the lower body may be sufficiently accurate but theareas around the neck may not be sufficiently accurate and may requireadditional data.

In some embodiments, the determination whether additional data is neededis based on input of the user. For example, the user is presented with arendered model compared to the images of the user. In the event the userdetermines the accuracy is insufficient, additional data is needed andprocessing proceeds to 203. In the event the user is satisfied with theaccuracy of the generated model, no additional data is needed andprocessing proceeds to 211.

At 211, the created three-dimensional model of the customer is appliedto one or more applications. In various embodiments, the applicationsare applied as described with respect to 105 of FIG. 1. For example, thecreated model can be used to determine a client vector of parametersthat represent the customer's body shape. Different purchasing and/ormatching results can be based on the client vector and/or clustering thecustomer's client vector with other customers.

FIG. 3 is a flow chart illustrating an embodiment of a process forcreating a custom three-dimensional model. Using the process of FIG. 3,a custom three-dimensional model of a customer is created from imagedata. In some embodiments, the process of FIG. 3 is performed at 103 ofFIG. 1 and/or at 207 of FIG. 2. In various embodiments, the process maybe performed on a device used to capture images of the user and/or aremote server.

At 301, image data is received. In various embodiments, the image datareceived corresponds to required poses necessary to model a user's bodyshape. The image data may include metadata such as lens data andorientation data. In some embodiments, the data includes athree-dimensional origin used to calibrate and orient each image withrespect to others in the sequence. In some embodiments, the imagesreceived have been processed at 205 of FIG. 2.

At 303, joints in the image data are annotated. Using the image datareceived at 301, the joints are annotated in the image. In variousembodiments, the joints are identified by selecting particular poses forthe images and/or isolating the movement at joints between images. Insome embodiments, the joints are identified in part by utilizing thecolor and/or depth channels of the images. In various embodiments, atrained machine learning model is used to identify joints using theimage data as input. In some embodiments, the joints are annotated by ahuman. In some embodiments, the joints in the image data are assigned atwo-dimensional location. The two-dimensional location may beextrapolated to a three-dimensional location with additional image data,such as image data showing the joints from different profiles.

At 305, the image data is segmented. In some embodiments, the image datais segmented to separate the foreground from the background. Theforeground can be identified using depth information and/or using colorchannels. In some embodiments, an image of the background without theuser is utilized to help identify the background from the foreground. Invarious embodiments, a trained machine learning model is used to segmentthe foreground and background. In some embodiments, the image data issegmented in parallel with joint annotations.

At 307, silhouettes are determined from the foreground data. In someembodiments, the segmented foregrounds are used to determinecorresponding silhouettes. For example, each segmented foreground isconverted to a silhouette. In various embodiments, a threshold value isused to convert the foreground into a black and white silhouette. Forexample, values that exceed the threshold value are included in thesilhouette and are colored white. Values that do not exceed thethreshold value are not included in the silhouette and are coloredblack.

At 309, a three-dimensional model is fit to the image data. Using theannotated joint and silhouette data, a predetermined three-dimensionalhuman model is modified to fit the joint and silhouette data. In variousembodiments, the fit model represents the body shape of the usercaptured in the image data.

FIG. 4 is a flow chart illustrating an embodiment of a process forfitting a three-dimensional model to a body shape. Using joint andsilhouette data from captured images, a predetermined three-dimensionalhuman model is modified until it matches the body shape of a user. Insome embodiments, the model is modified by using joints as structuralreference points and using silhouettes of the model to refine themodel's body composition. The joints and their relative positions helpto define the overall structure of the model. The model may be modifiedby modifying one of a set of parameters that adjust the predeterminedmodel. The parameters expand or contract the predetermined model,increasing or decreasing the effective body mass of the model indifferent areas. In some embodiments, one of a group of predeterminedthree-dimensional human models is used as a starting model. For example,a predetermined three-dimensional human model may be used for each sex.As another example, a different predetermined three-dimensional humanmodel may be used for children. In some embodiments, the process of FIG.4 is performed at 103 of FIG. 1, at 207 of FIG. 2, and/or at 309 of FIG.3. In various embodiments, the process may be performed on a device usedto capture images of the user and/or a remote server.

At 401, joints of the model are fit to poses. In the some embodiments, apredetermined three-dimensional human model includes multiple joints,such as joints for the shoulders, elbows, wrists, hands, fingers, hips,knees, ankles, feet, toes, neck, spine, head, etc. The three-dimensionallocation of the joints in the model can be modified to change theoverall structure of the model. For example, the legs of a model can bemade longer by increasing the distance between the hip and knee jointsas well as increasing the distance between the knee and ankle joints. Asanother example, a model can be made wider by increasing the distancebetween the hip joints. The torso can be made narrower by decreasing thedistance between shoulder joints. In various embodiments, the joints ofthe model are fit to the joint information annotated from different poseimages. In some embodiments, the three-dimensional locations of jointsof the model are fit based on two-dimensional joint locations annotatedfrom different pose images. Using multiple poses allows for more precisefitting and a closer approximation of the user's relative jointpositioning. In some embodiments, the joint information is created at303 of FIG. 3.

Once the joints of the model match the joints of the image poses, themodel has a skeletal shape similar to the user. Although the jointlocations may match, the body mass of the model may not match the user'sbody mass. For example, a heavier user and a lighter user can both sharethe same or similar joint locations. In some scenarios, the user's bodycomposition changes over time. A user that has gained or lost weightstill retains the same relative joint locations but has a different bodyshape at each weight. As another example, users with the same or similarjoint locations can carry their mass in different locations, such as inthe chest, upper body, midsection, and/or lower body, etc. Matching thejoint locations achieves an initial fitting but the model may requireadditional refinement to match the user's current body composition. Insome embodiments, as part of fitting joints to the model, one or morefacial feature locations are identified as well. The facial featurelocations may include eye, mouth, nose, and/or ear locations, amongothers.

In some embodiments, the joints of the model are fit by supplying theimage poses to a trained machine learning model and predicting the jointlocations. The trained machine learning model may be trained using imagedata and annotated joints, such as annotated joint data from a processsimilar to that of step 303 of FIG. 3. In some embodiments, the jointlocations are predicted in the event there is limited capture data ofthe user.

At 403, the model is fit to shape components. Using image silhouettedata, the model is modified to adjust the silhouette of the model to fiteach image silhouette. In some embodiments, the model is manipulated tomatch the pose of the user in each image and silhouettes of the modelare rendered. Body areas corresponding to locations where the silhouetteof the model does not extend to cover an image silhouette are expandedto match the image silhouette. Similarly, body areas corresponding tolocations where the silhouette of the model extends past the imagesilhouette are contracted to match the image silhouette. In variousembodiments, different image silhouettes are used to refine the bodyshape of the model. For example, face-on poses typically provide toolittle information about the shape of the user's chest and front-facingmidsection. Additional silhouette data from the side profile of the usermay be required to accurately model the user's chest and midsection. Invarious embodiments, each image silhouette places additional constraintson the body shape of the model and helps to refine the accuracy of themodel. In some embodiments, silhouette information is created at 307 ofFIG. 3. In various embodiments, the steps of 401 for fitting pose jointsand 403 for fitting shape components may be performed in parallel,iteratively, or sequentially (as shown). For example, if performediteratively, processing may alternative between each of steps 401 and403. In some embodiments, the steps are performed sequentially and step401 is performed before step 403 (as shown) or, alternatively, step 403is performed before step 401 (not shown).

In some embodiments, a fixed number of feature parameters are needed toadjust the model to fit a large range of body shapes. The body shapeparameters are selected to efficiently model the expected range of userswith minimal functional overlap. In some scenarios, as few as tenfeature parameters are needed. Example feature parameters may looselycorrespond to a user's height, width, leg length, width at hips, etc. Asadditional customers with different body shapes are added, new featureparameters can be introduced to provide better coverage of diverse bodyshapes. For example, feature parameters may correspond to the user'supper body muscularity, lower body muscularity, body mass index, etc. Insome embodiments, feature parameters correspond to different largemuscle groups such as the back, shoulder, arm, leg, and/or trunkmuscles, among others. In some embodiments, feature parameters includeone or more parameters for accurately modeling the user's chest.

In some embodiments, the body shape of the model is adjusted in aniterative process. For example, the model may be modified andsilhouettes compared. Based on the accuracy of the match, the model maybe further modified again. An accuracy percentage may be determined foreach created model. The iterative process may continue until an accuracypercentage meets a threshold value, a maximum number of iterations isperformed, or another appropriate limitation is reached.

In some embodiments, body shape components are fit using a machinelearning model by supplying the image silhouettes to a trained machinelearning model and predicting the body shape feature parameters. Thetrained machine learning model may be trained using silhouette data andcorresponding body feature parameters. In some embodiments, thesilhouette data is derived using portions of the process of FIG. 3. Insome embodiments, the body shape components are fit by a human, such asthe user, using a graphical user interface.

In some embodiments, the predetermined three-dimensional human model isa spectral decomposition model. Using spectral decomposition approach, amodification to a single vertex impacts more than the single vertex butalso surrounding vertices and regions. Similar to a human body, pullingon a single point of the surface of the body impacts all the surroundingskin. Using a spectral decomposition model, silhouette modifications canmore quickly converge on a match and help the model retain the shape ofa human body. A modification to one vertex of the model impacts theentire region of the vertex in the same direction as the vertexmodification. Regions of the model are modified by adjusting only alimited number of vertices that can be identified by comparingsilhouettes.

At 405, the silhouette fit for the created model is evaluated. Oncejoint and shape components have been fit at 403, the generatedthree-dimensional human model should closely approximate the customer'sbody shape. To determine the accuracy of the model, a silhouette fit forthe created model is evaluated. For example, similar to the process at403, silhouettes of the generated model are rendered and compared to theimage silhouettes. An accuracy percentage may be determined for thecreated model and additional data may be required. In some embodiments,particular body regions are identified as needing more data. Forexample, more detail may be needed of the feet, the crotch area, aroundthe neck, etc. In the event additional data is required, the user may beprompted to capture more data (not shown). In some embodiments, the useris presented with the created model and the user evaluates whether themodel accurately reflects the user's body shape. In some embodiments,the step of 405 is optional.

At 407, model parameters are extracted. Using the createdthree-dimensional human model, model parameters are extracted to createa client vector. The model parameters may include the feature parametersfor generating the model. In some embodiments, the model parameters alsoinclude the joint locations. The extracted client vector is a set ofparameters that accurately represents the user's body shape.

In some embodiments, the client vector is used to generate a set ofclient measurements for use in clothing sizing. For example, the clientvector and/or model can be used to predict for the user the measurementsthat a tailor would have taken by measuring the actual body of the user.Instead, a virtual tape measure can be applied to the generated model topredict the same measurements. In some embodiments, the client vector isused to generate the point of measurements for the user.

FIG. 5 is a diagram illustrating an example configuration for capturingimages of a customer's body shape. In various embodiments, theconfiguration is utilized in the event a tripod or another capture setupis not utilized. Device 501 is placed against wall 503 at a configuredangle, such as 30 degrees. Device 501 includes a camera and is used tocapture images of user 505. In the example shown, user 505 is depictedby a representation of the user at the location the human user shouldstand. The camera of device 501 is oriented to face user 505 (notshown). Device 501 is set up to capture different images of user 505including images of different poses and/or profiles. In variousembodiments, an application on device 501 provides instructions, such asvoice instructions, to user 505 to assist in the setup and capture ofimage data. For example, the instructions may include where and how tostand (e.g., move closer, move farther, move to the left, move to theright, turn left, turn right, lift arms, etc.). The instructions mayexplain what poses to make, when to start a pose, when a pose iscomplete, etc. In some embodiments, the instructions may be based onimage data captured in real time by device 501, such as the distanceuser 505 is from device 501. In some embodiments, device 501 is computersystem 1500 of FIG. 15. In some embodiments, the images are captured aspart of step 101 of FIG. 1 and/or 203 of FIG. 2.

FIG. 6 is a diagram illustrating an example image captured for creatinga custom three-dimensional model of a customer. Image data 601 includesuser 603 and background 605. User 603 is in the foreground of image data601. In some embodiments, the pose of user 603 is determined to increasethe accuracy of the data used for generating a custom three-dimensionalmodel. In the example shown, user 603 has his arms stretched out andraised to his sides to maximize the visible body area and to preventocclusions, such as covering hip joint locations. In some embodiments,image data 601 is captured as part of step 101 of FIG. 1 and/or step 203of FIG. 2. In some embodiments, image data 601 is processed at 205 ofFIG. 2 and/or received at 301 of FIG. 3 as part of a series of imagedata of a customer. In some embodiments, image data 601 is extractedfrom captured video data. In some embodiments, image data 601 iscaptured using the configuration of FIG. 5.

FIG. 7 is a diagram illustrating an example image annotated with jointdata for creating a custom three-dimensional model of a customer. Imagedata 701 includes user 703 with specific joints annotated. Identifiedjoints, including joints 705 and 707, are depicted as filled circles.The lines connecting the joints represent a structural skeleton of user703. In the example shown, joints 705 are circled and point to wrist,elbow, and hip joints. Joint 707 is circled and is a misidentified jointpositioned above the shoulder of user 703. In various embodiments, asmore image data with corresponding additional constraints is annotated,the accuracy of the joint data and annotation improves. In someembodiments, joint data is annotated at 303 of FIG. 3. In someembodiments, image data 701 is a processed version of image data 601 ofFIG. 6.

FIG. 8 is a diagram illustrating an example of a segmented image usedfor creating a custom three-dimensional model of a customer. Image data801 includes user 803 and has had the background removed viasegmentation. User 803 is the foreground portion of an image andcorresponds to the user portion of the image. In various embodiments, asmore image data is captured, the accuracy of the segmentation improves.In the example shown, artifacts 805 are circled and point to twoportions of the image incorrectly identified as foreground portions.Similarly, areas around the head, areas outlining the arms, and areasbetween the legs near the crotch of user 803 include portions of thebackground that have been incorrectly included in the foreground. Insome embodiments, depth information is used to improve the segmentationprocess to more accurately remove these background portions such asartifacts 805. In some embodiments, image data is segmented at 305 ofFIG. 3. In some embodiments, image data 801 is a processed version ofimage data 601 of FIG. 6.

FIG. 9 is a diagram illustrating an example of a silhouette image usedfor creating a custom three-dimensional model of a customer. Silhouetteimage 901 includes the user silhouette 903 that corresponds to thesilhouette of the customer. In the example shown, user silhouette 903 iscolored in white and the rest of silhouette image 901 is black. In someembodiments, silhouette image 901 is image data 801 of FIG. 8 afterconverting the segmented foreground of user 803 of FIG. 8 into a usersilhouette 903. In some embodiments, a silhouette image is determined at307 of FIG. 3.

FIG. 10 is a diagram illustrating an embodiment of a three-dimensionalhuman model fit to joint data. In the example shown, model 1000 includesjoints, including joints 1001, depicted as circles. Prior to jointfitting, model 1000 was a predetermined three-dimensional human model.In the example shown, the three-dimensional locations of joints of model1000 are fit based on pose images such as image data 701 with annotatedtwo-dimensional joint location data. The location of the jointsincluding their relative locations to one another have been modified. Insome embodiments, joints 1001 point to wrist, elbow, and hip joints andare fit to joints 705 of FIG. 7. By modifying the joint locations, themodel more closely approximates the body shape of a user. In someembodiments, the joints of model 1000 are fit at 401 of FIG. 4.

FIG. 11A is a diagram illustrating an embodiment of a three-dimensionalhuman model compared to an image silhouette. The diagram of FIG. 11Arepresents the process of fitting a three-dimensional model to atwo-dimensional silhouette. In the example shown, model 1101 is athree-dimensional human model and image silhouette 1103 is atwo-dimensional silhouette converted from image data. In someembodiments, model 1101 is model 1000 of FIG. 10 and silhouette 1103 isextracted from silhouette image 901 of FIG. 9. In various embodiments,the shape components of model 1101 are modified to fit the silhouette ofmodel 1101 to image silhouette 1103. In some embodiments, the process tofit the shape components of the model is performed at 403 of FIG. 4.

FIG. 11B is a diagram illustrating an embodiment of a process formodifying a three-dimensional human model to fit an image silhouette. Inthe example shown, model 1151 is a predetermined three-dimensional humanmodel. Model 1151 represents a model with no parameter modifications. Inorder to create a custom model, model 1151 is modified to match the bodyshape of a user. In the example shown, model 1153 is a model with afeature parameter corresponding to height adjusted to increase theheight of model 1151. Although model 1153 is taller than model 1151,model 1153 is still too short compared to the user. Compared to model1153, model 1155 is a model with a feature parameter corresponding toheight that matches an image silhouette of the user. In someembodiments, the process of fitting a model to an image silhouette is aniterative process illustrated by the sequence of models starting withmodel 1151, made taller with model 1153, and finishing with model 1155.In various embodiments, the model parameter modified as illustrated inFIG. 11B is only one of the parameters that can be modified to best fitthe shape components of a model to the body shape of the user. Moreover,the models of FIG. 11B are only representative models to illustrate theprocess and the actual iterative process may include many more iterativemodels and changes to many more shape component parameters. In someembodiments, the process to fit the shape components of the model isperformed at 403 of FIG. 4.

FIG. 12 is a diagram illustrating an embodiment of a process for fittinga three-dimensional human model to a customer body shape. In the exampleshown, model 1201 is a custom three-dimensional human model being fit toa customer's body shape. In some embodiments, the process of fitting amodel is performed using the process of FIG. 4. Identified joints,including joints 1203, of model 1201 are circled. In the example shown,joints 1203 point to wrist, elbow, and hip joints. In some embodiments,model 1201 corresponds to model 1000 of FIG. 10 and joints 1203 arecorrespond to joints 1001 of FIG. 10. In some embodiments, the joints ofmodel 1201 are fit at 401 of FIG. 4. In the example shown, the pose ofmodel 1201 is articulated to mimic the pose of silhouette 1205.Silhouette 1205 is generated from a silhouette of the customer such asfrom user silhouette 903 of silhouette image 901 of FIG. 9. To determinehow to modify model 1201 to more closely match the body shape of thecustomer, the body shape of model 1201 is compared to silhouette 1205.In some embodiments, the process for fitting model 1201 to silhouette1205 is performed at 405 of FIG. 4. In the example shown, additional fitrefinement is needed since silhouette 1205 and model 1201 are not anexact match. In certain areas, silhouette 1205 protrudes from theoutline of model 1201.

FIG. 13 is a diagram illustrating an embodiment of a graphical userinterface (GUI) for creating a three-dimensional human model of acustomer. Using the GUI of FIG. 13, a user can create and/or review acreated model of the customer. In some embodiments, the user may alsomodify the created model to adjust any body portions that requireadditional refinement. In the example shown, device 1301 displays modelrendering 1303. Device 1301 may be a smartphone device, a tablet, oranother similar device. Model rendering 1303 includes a rendering of thecustom three-dimensional model of the customer (in dots) overlaid on animage captured of the user. By viewing model rendering 1303, a user canvisually inspect the accuracy of the model. In some embodiments, theuser can accept or reject the created model. In various embodiments, themodel rendered in model rendering 1303 is created using one or more ofthe processes of FIGS. 1-4.

In some embodiments, a user of device 1301 can select portions of modelrendering 1303 to identify areas of the model that require additionalrefinement. For example, a user can select the rendering of the model'sright foot to indicate that the modeling of the customer's right footrequires additional refinement. In some embodiments, the user canmanipulate the model to refine the accuracy of the model. For example,the user may rotate the model, articulate the model at joint locations,expand or contract portions of the model to add or remove body mass,etc. As another example, the user can lengthen or shorten the modeland/or widen or narrow the model. In some embodiments, model rendering1303 is displayed on a touchscreen display and the user can manipulatethe model using touch gestures. In some embodiments, device 1301 isdevice 501 of FIG. 5 and/or computer system 1500 of FIG. 15. In someembodiments, the interface of FIG. 13 is used at 207, 209, and/or 211 ofFIG. 2 as part of the process for creating the custom model for acustomer.

FIG. 14 is a diagram illustrating an embodiment of a graphical userinterface (GUI) for modifying body shape parameters of athree-dimensional human model. In the example shown, user interface 1401can be used to adjust multiple feature parameters that modify a model'sbody shape. User interface 1401 includes parameters: height, width, armlength, hips, and torso that can be adjusted using sliders. The sliderfor torso displays a numeric value associated with the torso parameter.Although five parameters are shown with slider controls, in variousembodiments, additional (or fewer) parameters may be modified usingappropriate user interface controls. In some embodiments, as theparameters for the model are modified, a rendering of the model, such asmodel rendering 1303 of FIG. 13, is displayed in real-time to update theuser on the impact of the changes. In some embodiments, the valuesassociated with the feature parameters are a client vector and representa body profile of a customer. In some embodiments, the user interface ofFIG. 14 is used at 103 of FIG. 1 and/or at 207 and/or 209 of FIG. 2 forassisting in the creation of a three-dimensional human model of acustomer. In various embodiments, the GUI of FIG. 14 is displayed on adevice such as device 501 of FIG. 5 and/or computer system 1500 of FIG.15.

FIG. 15 is a functional diagram illustrating a programmed computersystem for creating a custom three-dimensional model of a customer. Forexample, a programmed computer system may be a mobile device, such as asmartphone device, a tablet, a kiosk, a laptop, a smart television,and/or another similar device for capturing images of a user. In someembodiments, a programmed computer system is a desktop or servercomputer system for processing captured image data and creating a customthree-dimensional model of a customer. As will be apparent, othercomputer system architectures and configurations can be used. Computersystem 1500, which includes various subsystems as described below,includes at least one microprocessor subsystem (also referred to as aprocessor or a central processing unit (CPU)) 1501. For example,processor 1501 can be implemented by a single-chip processor or bymultiple processors. In some embodiments, processor 1501 is a generalpurpose digital processor that controls the operation of the computersystem 1500. Using instructions retrieved from memory 1503, theprocessor 1501 controls the reception and manipulation of input data,and the output and display of data on output devices (e.g., display1511). In some embodiments, processor 1501 includes and/or is used toprovide functionality for creating custom three-dimensional models ofcustomers. In some embodiments, processor 1501 is used to perform atleast part of the processes described with respect to FIGS. 1-4 andillustrated with respect to the diagrams of FIGS. 6-14. In someembodiments, computer system 1500 is device 501 of FIG. 5.

Processor 1501 is coupled bi-directionally with memory 1503, which caninclude a first primary storage, typically a random access memory (RAM),and a second primary storage area, typically a read-only memory (ROM).As is well known in the art, primary storage can be used as a generalstorage area and as scratch-pad memory, and can also be used to storeinput data and processed data. Primary storage can also storeprogramming instructions and data, in the form of data objects and textobjects, in addition to other data and instructions for processesoperating on processor 1501. Also as is well known in the art, primarystorage typically includes basic operating instructions, program code,data, and objects used by the processor 1501 to perform its functions(e.g., programmed instructions). For example, memory 1503 can includeany suitable computer-readable storage media, described below, dependingon whether, for example, data access needs to be bi-directional oruni-directional. For example, processor 1501 can also directly and veryrapidly retrieve and store frequently needed data in a cache memory (notshown).

A removable mass storage device 1507 provides additional data storagecapacity for the computer system 1500, and is coupled eitherbi-directionally (read/write) or uni-directionally (read only) toprocessor 1501. For example, removable mass storage device 1507 can alsoinclude computer-readable media such as flash memory, portable massstorage devices, magnetic tape, PC-CARDS, holographic storage devices,and other storage devices. A fixed mass storage 1505 can also, forexample, provide additional data storage capacity. Common examples ofmass storage 1505 include flash memory, a hard disk drive, and an SSDdrive. Mass storages 1505, 1507 generally store additional programminginstructions, data, and the like that typically are not in active use bythe processor 1501. Mass storages 1505, 1507 may also be used to storeuser-generated content and digital media for use by computer system1500. It will be appreciated that the information retained within massstorages 1505 and 1507 can be incorporated, if needed, in standardfashion as part of memory 1503 (e.g., RAM) as virtual memory.

In addition to providing processor 1501 access to storage subsystems,bus 1510 can also be used to provide access to other subsystems anddevices. As shown, these can include a network interface 1509, a display1511, a touch-screen input device 1513, a camera 1515, additionalsensors 1517, additional output generators 1519, as well as an auxiliaryinput/output device interface, a sound card, speakers, additionalpointing devices, and other subsystems as needed. For example, anadditional pointing device can be a mouse, stylus, track ball, ortablet, and is useful for interacting with a graphical user interface.In the example shown, display 1511 and touch-screen input device 1513may be utilized for displaying a graphical user interface for capturingimages of a customer and/or creating/modifying a customthree-dimensional model of the customer. In some embodiments, camera1515 and/or additional sensors 1517 include a depth sensor for capturingdepth information along with image data.

The network interface 1509 allows processor 1501 to be coupled toanother computer, computer network, telecommunications network, ornetwork device using one or more network connections as shown. Forexample, through the network interface 1509, the processor 1501 cantransmit/receive captured images of a customer and/or a created customthree-dimensional model of the customer. Further, through the networkinterface 1509, the processor 1501 can receive information (e.g., dataobjects or program instructions) from another network or outputinformation to another network in the course of performingmethod/process steps. Information, often represented as a sequence ofinstructions to be executed on a processor, can be received from andoutputted to another network. An interface card or similar device andappropriate software implemented by (e.g., executed/performed on)processor 1501 can be used to connect the computer system 1500 to anexternal network and transfer data according to standard protocols. Forexample, various process embodiments disclosed herein can be executed onprocessor 1501, or can be performed across a network such as theInternet, intranet networks, or local area networks, in conjunction witha remote processor that shares a portion of the processing. In someembodiments, network interface 1509 utilizes wireless technology forconnecting to networked devices such as device 501 of FIG. 5. In someembodiments, network interface 1509 utilizes a wireless protocoldesigned for short distances with low-power requirements. In someembodiments, network interface 1509 utilizes a version of the Bluetoothprotocol. Additional mass storage devices (not shown) can also beconnected to processor 1501 through network interface 1509.

An auxiliary I/O device interface (not shown) can be used in conjunctionwith computer system 1500. The auxiliary I/O device interface caninclude general and customized interfaces that allow the processor 1501to send and, more typically, receive data from other devices such asmicrophones, touch-sensitive displays, transducer card readers, tapereaders, voice or handwriting recognizers, biometrics readers, cameras,portable mass storage devices, and other computers.

In addition, various embodiments disclosed herein further relate tocomputer storage products with a computer readable medium that includesprogram code for performing various computer-implemented operations. Thecomputer-readable medium is any data storage device that can store datawhich can thereafter be read by a computer system. Examples ofcomputer-readable media include, but are not limited to, all the mediamentioned above and magnetic media such as hard disks, floppy disks, andmagnetic tape; optical media such as CD-ROM disks; magneto-optical mediasuch as optical disks; and specially configured hardware devices such asapplication-specific integrated circuits (ASICs), programmable logicdevices (PLDs), and ROM and RAM devices. Examples of program codeinclude both machine code, as produced, for example, by a compiler, orfiles containing higher level code (e.g., script) that can be executedusing an interpreter.

The computer system shown in FIG. 15 is but an example of a computersystem suitable for use with the various embodiments disclosed herein.Other computer systems suitable for such use can include additional orfewer subsystems. In addition, bus 1510 is illustrative of anyinterconnection scheme serving to link the subsystems. Other computerarchitectures having different configurations of subsystems can also beutilized.

Although the foregoing embodiments have been described in some detailfor purposes of clarity of understanding, the invention is not limitedto the details provided. There are many alternative ways of implementingthe invention. The disclosed embodiments are illustrative and notrestrictive.

What is claimed is:
 1. A method, comprising: receiving a series ofcaptured images of a user; extracting one or more silhouettes from theseries of captured images of the user; using a processor to process thecaptured images to identify a portion of each of the captured imagescorresponding to the user, wherein the identified portions of thecaptured images corresponding to the user include identified user jointlocations of the user; modifying, based on the identified user jointlocations of the user, parameters of a predetermined three-dimensionalhuman model to fit a modified version of the predeterminedthree-dimensional human model across the identified portions of thecaptured images to determine a set of specific parameters representing abody profile of the user; and refining the modified version of thepredetermined three-dimensional human model based on comparing themodified version of the predetermined three-dimensional human model tothe one or more extracted silhouettes.
 2. The method of claim 1, whereinthe parameters of the predetermined three-dimensional human modelinclude one or more of the following: a height, a width, a leg length,and a hip width parameter.
 3. The method of claim 1, wherein the seriesof captured images of the user includes color data and depth data. 4.The method of claim 1, wherein the identified portions of the capturedimages corresponding to the user include identified foreground portionsof the captured images.
 5. The method of claim 1, wherein the user jointlocations include shoulder, elbow, wrist, hand, hip, knee, ankle, andfoot locations.
 6. The method of claim 1, wherein the identifiedportions of the captured images corresponding to the user includeidentified facial feature locations of the user.
 7. The method of claim6, wherein the identified facial feature locations include earlocations, eye locations, a nose location, and a mouth location.
 8. Themethod of claim 1, further comprising determining a value identifying apredicted size fit between an item and the user based on the set ofspecific parameters representing the body profile of the user.
 9. Themethod of claim 8, further comprising selecting and providing the itemto the user based on the value identifying the predicted size fitexceeding a threshold value.
 10. The method of claim 1, wherein thepredetermined three-dimensional human model is based on a sex or an ageof the user.
 11. The method of claim 1, wherein the predeterminedthree-dimensional human model includes a spectral decomposition model.12. The method of claim 1, further comprising: identifyingtwo-dimensional joint locations of the user in each of the capturedimages; and fitting three-dimensional joint locations of thepredetermined three-dimensional human model based on the identifiedtwo-dimensional joint locations.
 13. The method of claim 1, furthercomprising determining an accuracy metric of the modified version of thepredetermined three-dimensional human model including by comparingrendered silhouettes of the modified version to silhouette images basedon the identified portions of the captured images corresponding to theuser.
 14. The method of claim 1, further comprising matching a stylistto the user based on attributes of the stylist matching the set ofspecific parameters representing the body profile of the user.
 15. Themethod of claim 1, further comprising: selecting a subset of parametersof the set of specific parameters representing the body profile of theuser; and identifying a group of users with similar body profiles usingthe selected subset of parameters.
 16. The method of claim 15, furthercomprising at least in part automatically determining a parameter of agarment sized for the identified group of users.
 17. The method of claim1, further comprising: displaying a rendering of the modified version ofthe predetermined three-dimensional human model to the user; andreceiving feedback from the user regarding an accuracy rating of themodified version.
 18. The method of claim 17, wherein the feedbackidentifies a portion of the modified version where an additionalrefinement is desired.
 19. The method of claim 18, further comprisingreceiving one or more additional captured images of the usercorresponding to the portion of the modified version where theadditional refinement is desired.
 20. A computer program product, thecomputer program product being embodied in a non-transitory computerreadable storage medium and comprising computer instructions for:receiving a series of captured images of a user; extracting one or moresilhouettes from the series of captured images of the user; processingthe captured images to identify a portion of each of the captured imagescorresponding to the user, wherein the identified portions of thecaptured images corresponding to the user include identified user jointlocations of the user; modifying, based on the identified user jointlocations of the user, parameters of a predetermined three-dimensionalhuman model to fit a modified version of the predeterminedthree-dimensional human model across the identified portions of thecaptured images to determine a set of specific parameters representing abody profile of the user; and refining the modified version of thepredetermined three-dimensional human model based on comparing themodified version of the predetermined three-dimensional human model tothe one or more extracted silhouettes.
 21. A system, comprising: aprocessor; and a memory coupled with the processor, wherein the memoryis configured to provide the processor with instructions which whenexecuted cause the processor to: receive a series of captured images ofa user; extract one or more silhouettes from the series of capturedimages of the user; process the captured images to identify a portion ofeach of the captured images corresponding to the user, wherein theidentified portions of the captured images corresponding to the userinclude identified user joint locations of the user; and modify, basedon the identified user joint locations of the user, parameters of apredetermined three-dimensional human model to fit a modified version ofthe predetermined three-dimensional human model across the identifiedportions of the captured images to determine a set of specificparameters representing a body profile of the user; and refine themodified version of the predetermined three-dimensional human modelbased on comparing the modified version of the predeterminedthree-dimensional human model to the one or more extracted silhouettes.